By comprehensively comparing the models of traditional knowledge graph representation learning, including the advantages and disadvantages and the applicable tasks, the analysis shows that the traditional single-modal knowledge graph cannot represent knowledge well. Therefore, how to use multimodal data such as text, image, video, and audio for knowledge graph representation learning has become an important research direction. At the same time, the commonly used multimodal knowledge graph datasets were analyzed in detail to provide data support for relevant researchers. On this basis, the knowledge graph representation learning models under multimodal fusion of text, image, video, and audio were further discussed, and various models were summarized and compared. Finally, the effect of multimodal knowledge graph representation on enhancing classical applications, including knowledge graph completion, question answering system, multimodal generation and recommendation system in practical applications was summarized, and the future research work was prospected.
In order to reduce the regression test set and improve the efficiency of regression test in the Continuous Integration (CI) environment, a regression test suite selection method for the CI environment was proposed. First, the commits were prioritized based on the historical failure rate and execution rate of each test suite related to each commit. Then, the machine learning method was used to predict the failure rates of the test suites involved in each commit, and the test suite with the higher failure rate were selected. In this method, the commit prioritization technology and the test suite selection technology were combined to ensure the increase of the failure detection rate and the reduction of the test cost. Experimental results on Google’s open-source dataset show that compared to the methods with the same commit prioritization method and test suite selection method, the proposed method has the highest improvement in the Average Percentage of Faults Detected per cost (APFDc) by 1% to 27%; At the same cost of test time, the TestRecall of this method increases by 33.33 to 38.16 percentage points, the ChangeRecall increases by 15.67 to 24.52 percentage points, and the test suite SelectionRate decreases by about 6 percentage points.
In order to solve the problem of low matching quality of the traditional feature extraction and matching algorithm in Unmanned Aerial Vehicle (UAV) visual localization, a new image matching algorithm based on transmission tower area extraction was proposed. Firstly, the image was divided into several overlapping grid areas, and the feature points were extracted by a two-layer pyramid structure for each area to ensure the uniform distribution of feature points. Then, the Line Segment Detector (LSD) algorithm was used to extract the lines in the images, the transmission tower support areas were extracted on the basis of special structure of transmission tower. Finally, the feature points in the transmission tower areas and the background areas were matched respectively in continuous images to further estimate the camera motion. In the rotation and translation estimation experiment, compared with the traditional Oriented Features from Accelerated Segment Test(FAST) and Rotated Binary Robust Independent Elementary Features (BRIEF) (ORB) feature extraction and matching algorithm, the proposed algorithm has the feature matching accuracy improved by 10.1 percentage points, the mean value of relative pose error reduced by 0.049. In the UAV inspection experiment, the relative error of the UAV trajectory estimation by using the proposed algorithm is 2.89%, which indicates that the proposed algorithm can achieve the robust and accurate estimation of the UAV’s pose during the real-time flying around the tower.
Influence maximization is one of the important issues in social network analysis, which aims to identify a small group of seed nodes. When these nodes act as initial spreaders, information can be spread to the remaining nodes as much as possible in the network. The existing heuristic algorithms based on network topology usually only consider one single network centrality, failing to comprehensively combine node characteristics and network topology; thus, their performance is unstable and can be easily affected by the network structure. To solve the above problem, an influence maximization algorithm based on Node Coverage and Structural Hole (NCSH) was proposed. Firstly, the coverages and grid constraint coefficients of all nodes were calculated. Then the seed was selected according to the principle of maximum coverage gain. Secondly, if there were multiple nodes with the same gain, the seed was selected according to the principle of minimum grid constraint coefficient. Finally, the above steps were performed repeatedly until all seeds were selected. The proposed NCSH maintains good performance on six real networks under different numbers of seeds and different spreading probabilities. NCSH achieves 3.8% higher node coverage than to the similar NCA (Node Coverage Algorithm) on average, and 43% lower time consumption than the similar SHDD (maximization algorithm based on Structure Hole and DegreeDiscount). The experimental results show that the NCSH can effectively solve the problem of influence maximization.
In order to solve the occlusion problem of student expression recognition in complex classroom scenes, and give full play to the advantages of deep learning in the application of intelligent teaching evaluation,a student expression recognition model and an intelligent teaching evaluation algorithm based on deep attention network in classroom teaching videos were proposed. A video library, an expression library and a behavior library for classroom teaching were constructed, then, multi-channel facial images were generated by cropping and occlusion strategies. A multi-channel deep attention network was built and self-attention mechanism was used to assign different weights to multiple channel networks. The weight distribution of each channel was restricted by a constrained loss function, then the global feature of the facial image was expressed as the quotient of the sum of the product of the feature times its attention weight of each channel divided by the sum of the attention weights of all channels. Based on the learned global facial feature, the student expressions in classroom were classified, and the student facial expression recognition under occlusion was realized. An intelligent teaching evaluation algorithm that integrates the student facial expressions and behavior states in classroom was proposed, which realized the recognition of student facial expressions and intelligent teaching evaluation in classroom teaching videos. By making experimental comparison and analysis on the public dataset FERplus and self-built classroom teaching video datasets, it is verified that the student facial expressions recognition model in classroom teaching videos achieves high accuracy of 87.34%, and the intelligent teaching evaluation algorithm that integrates the student facial expressions and behavior states in classroom achieves excellent performance on the classroom teaching video dataset.
In view of the problems that classroom teaching scene is obscured seriously and has numerous students, the current video action recognition algorithm is not suitable for classroom teaching scene, and there is no public dataset of student classroom action, a classroom teaching video library and a student classroom action library were constructed, and a real-time multi-person student classroom action recognition algorithm based on deep spatiotemporal residual convolution neural network was proposed. Firstly, combined with real-time object detection and tracking to get the real-time picture stream of each student, and then the deep spatiotemporal residual convolution neural network was used to learn the spatiotemporal characteristics of each student’s action, so as to realize the real-time recognition of classroom behavior for multiple students in classroom teaching scenes. In addition, an intelligent teaching evaluation model was constructed, and an intelligent teaching evaluation system based on the recognition of students’ classroom actions was designed and implemented, which can help improve the teaching quality and realize the intelligent education. By making experimental comparison and analysis on the classroom teaching video dataset, it is verified that the proposed real-time classroom action recognition model for multiple students in classroom teaching video can achieve high accuracy of 88.5%, and the intelligent teaching evaluation system based on classroom action recognition has also achieved good results in classroom teaching video dataset.